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PATTERN AND TEXTURE ORIENTATION ESTIMATOR 



Technical Field of the Invention 

The present invention relates generally to image processing and, in particular, to 
5 the automatic estimation of patf^eJM- ^iJ®i:^ture orientation of an im^ge. 



Background Art 

Images are often characterised by a number of features including pattern and 
texture. A number of image processing operations depend upon the estimation of the 
10 orientation of these image features. 

Image processing/analysis applications where the estimation of orientation is 
important include: 

• edge orientation detection; 

• orientation input to steerable filters; 
15 • pattern analysis; 

• texture analysis, where the local orientation of texture is estimated for 
characterisation purposes; and 

• orientation selective filtering, processing, and enhancement of images. 

One such area where, the estimation of orientation is particularly important is the 
20 demodulation of fringe patterns or equivalently, AM-FM modulated patterns. Examples 
of such patterns are illustrated in Figs. 2 A and 5 A. One area where such patterns exist is 
in fingerprint analysis. Other areas include the analysis of naturally formed patterns such 
as an iris diaphragm, ripples in sand, and fluid convection in thin layers. The primary 
reason why it is useful to know the fringe orientation when demodulating such a fringe 
25 pattern is because it allows for I -dimensional demodulators to be aligned correctly. 

Another area where the estimation of pattern orientation is used is in image re- 
sampling and enhancement. 

Known metliods for estimating pattern orientation include gradient-based 
methods. A serious deficiency with such gradient-based methods is that at the ridges and 
30 valleys of images, the gradient has zero magnitude. Gradient methods are therefore 
unable to provide information about pattern direction in such regions. 
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It is an object of the present invention to substantially overcome, or at least 
ameliorate, one or more disadvantages of existing arrangements. 

According to a first aspect of the invention, there is provided a method of 
estimating an orientation angle of a pattern in an image, said method comprising the steps 
5 of: 

applying a complex energy operator to the image to provide an energy encoded 

image; 

determining a phase component of said energy encoded image; and 
calculating said orientation angle from said phase component of said energy 
10 encoded image. 

According to another aspect of the invention, there is provided an apparatus for 
implementing any one of the aforementioned method. 

Brief Description of the Drawings 

15 A preferred embodiment of the present invention will now be described with 

reference to the drawings, in which: 

Fig. 1 is an illustration of the local structure of a fringe pattern; 
Fig. 2A is an example of a fringe pattern image upon which the preferred 
embodiment of the invention can be appHed; 
20 Figs. 2B and 2C are the magnitude and phase components respectively of an 

intermediate result in applying an energy operator to the image of Fig. 2A; 

Figs. 2D and 2E are the magnitude and phase components respectively of 
another intermediate result in applying the energy operator to the image of Fig. 2A; 

Figs. 2F and 2G are the magnitude and phase componenls^ respectively of 
25 applying the energy operator to the image of Fig. 2A; 

Fig. 3 A is an example of a fringe pattern image containing noise, upon which the 
preferred embodiment of the invention can be applied; 

Figs. 3B and 3C are the magnitude and phase components respectively of 
applying the energy operator to the image of Fig. 3 A; 
30 Figs. 3D and 3E are the magnitude and phase components respectively of 

applying a modified energy operator of the preferred embodiment of the invention to the 
image of Fig. 3A; 

Fig. 4A is an example of a fringe pattern image containing noise, upon which the 
preferred embodiment of the invention can be applied; 
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Figs. 4B and 4C are the magnitude and phase components respectively of 
applying the energy operator to the image of Fig. 4A; 

Figs. 4D and 4E are the magnitude and phase components respectively of 
applying the modified energy operator of the preferred embodiment of the invention to 
the image of Fig. 4A; 

Fig. 5A is an example of a fringe pattern image with amplitude and frequency 
modulation, upon which the preferred embodiment of the invention can be applied; 

Figs. 5B and 5C are the magnitude and phase components respectively of 
applying the energy operator to the image of Fig. 5 A; 

Figs. 5D and 5E are the magnitude and phase components respectively of 
applying the modified energy operator of the preferred embodiment of the invention to 
the image of Fig. 5 A; 

Fig. 6A is an example of a texture image, upon which the preferred embodiment 
of the invention can be appHed; 

Figs. 6B and 6C are the magnitude and phase components respectively of 
applying the energy operator to the image of Fig. 6A; 

Figs. 6D and 6E are the magnitude and phase components respectively of 
applying the modified energy operator of the preferred embodiment of the invention to 
the image of Fig. 6A; 

Fig. 7 is a schematic block diagram of a general purpose computer upon which 
an embodiment of the present invention can be practised; and 

Fig. 8 is a flow diagram of a method according to the preferred embodiment of 
the invention. 

Detailed Description including Best Mode 
In recent years, an efficient method of signal demodulation using a concept 
known as an "energy operator" has become popular. The operator can be used for 
demodulation of AM and FM signals, including combined AM-FM signals, using a result 
from the analysis of simple harmonic motion (SHM). The energy operator 4^ is defined 
as: 
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A particularly useful property of the energy operator 4^ is that it gives uniform 
estimates of modulation parameters anywhere on the signal / Essentially this is due to 
the fact that, at local peaks (and troughs) of the signal /, the energy is concentrated in the 
potential energy component, represented by the second term in Equation (1), whereas at 
5 the zero crossings, the energy is in the kinetic component represented by the first term in 
Equation (1). In other words, the rapidly varying sine and cosine components associated 
with AM-FM signals are exactly cancelled out by the energy operator leaving an 
underlying constant representing total energy. 

The operation of the energy operator has also been extended to N-dimensions. 
10 This is described in the publication "A multi-dimensional energy operator for image 
processing", SPIE Conference on Visual Communications and Image Processing, Boston, 
MA, [1992], 177-186, by P. Maragos, A.C. Bovik, and J. F. Quartieri, the contents of 
which are incorporated herein by cross reference. 

Utilising the energy operator T on images, (i.e. data sets formed in 2-dimensions,) 
15 separate x and y components may be calculated and added together to obtain an overall 2- 
dimensional energy operator. 

However, the energy operator ^ defined in this simple way has certain deficiencies. 
The main deficiency being that the orientation associated with the energy components is 
strictly limited to the positive quadrant of the x-y plane, which means that visually very 
20 different orientations are indistinguishable using two positive energy components. 

Referring to Fig. 1, consider a 2-dimensional fringe pattern or AM-FM pattern, 
located around a point (xo^yo) and formed by a number of fringes 10. The intensity of the 
fringe pattern can be represented as follows: 

fix,y) - h(x,y).cos[2n(uo{x-xo}+vo{y-yo})^z] (2) 

25 wherein any slowly varying (non-multiplicative) background offsets are removed 

with an appropriate high-pass filter. An amplitude modulation term is denoted by b(x,y), 
which is assumed to be a slowly varying function. A small region of interest is defined 
by Q. Within this small region Q, the spatial frequency components uq and vq are slowly 
varying so that they are effectively constant around the point (xo.yo), making the carrier a 

30 linear function of x and 3^. A normal 12 to the local fringes is at an angle fi^ to the x-axis. 
The local spatial frequency components uq and vq also satisfy the following Equations: 

"0 + = ^0 (3) 
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_2^ 

= A, (4) 



"o . ^ -' 

wherein Xq is the fringe spacing. 

The X and >^ component energy operators and % respectively may be performed 
on the fringe pattern defined in Equation (1) as follows: 

1',{/(A^.y)) = (fJ-/^=.(2/r„,*)^ (6) 



>F,(/(.,.)) = (|J-/|f.(2.v.6,' 

Substituting Equations (6) and (7) into Equations (3) and (5), the local frequency 
and orientation of the local fringe pattern may alternatively be expressed as follows: 



(8) 



Equation (9) indicates a loss in sign information, which corresponds to an inabiHty 
to distinguish between fringes at four different orientations fi^.K (i^^n - (i^. This 

is very undesirable because the human eye can clearly separate (i^ frorf! ^ - /?o , and -(i^ 
from 71 + In general, however, it is not possible to distinguish (on a local level at 
least) between any orientation and the tt rotated orientation. 

An alternative approach to extending the energy operator to two dimensions is by 
defming a two dimensional complex operator H^^ which encodes the energy in the 
magnitude and the orientation in the phase: 

^..{/} + ^.{/} = (2^6)^K+/v,^) (10) 
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As mentioned above, the simple x and y components alone fail to represent the 
orientation correctly. A preferable approach would be to have a 2D energy operator that 
has the following output: 

{/} = {27ib)' {u, + iv,)' = {2KCT,b)' exp(2zA) (11) 

The complex energy operator enables the isolation of the fringe spacing Xq, as 
seen in Fig. 1, and orientation angle po respectively. It is noted that the term 2 in the 
exponential factor of Equation (11) again causes the orientation angle Po to be defined 
modulo n. Such an energy operator is developed by defining a 2- dimensional complex 
differential operator D as follows: 

The complex energy operator can then be defined as follows: 

{D{f}Y - fD'{f] = {2ncT,by ^xp{2ip,) = ^^^{f] (13) 

With the Fourier transform defined as follows: 

03 00 

P'{u,v)= I \f{x, >')exp[- 2m{ux + ^y)\lxdy ( 1 4) 

—00— 00 

^5 the operator D can be implemented in the Fourier space , providing the following 

relationship: 

{2m\u + iv)F {u,v)^ ^ ^D{f{x,y)}Qxp[-27d{ux + vy)\lxdy (15) 
wherein 

27z:(w + /v)^ 27r^7.exp(/^) (16) 
20 and the polar coordinate transforms are: 



u ~c/cos((p) 
v = qsin{(p) 



(17) 



The corresponding Fourier operation of Equation (15) provides for the 
multiplication of the Fourier transform of the fringe pattern F(u,vJ with a spiral phase 
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filter, gexp(/^). The spiral phase filter has the characteristic of linearly (conical) 
increasing in magnitude with fi^equency. The Fourier transformation pairs may be written 
as: 

F{u,v)<^ f{x,y) (18) 
5 {2m)qoxp{i(p)F{u,v)<:^D{f(x,y)} (19) 

{2myq'cxp{2i(p)F{u,v)^D^{f(x,y)} (20) 

The derivatives in Equation (13) can therefore be implemented as 2-dimensional 
Fourier filters with spiral phase and conical magnitude. 

By applying the differential operators defined in Equations (18) to (20) as filters 

10 through simple multiplication of the image of Fig. 2 A, the results as shown in Figs. 2B to 
2G is obtained. Figs. 2B and 2C show the magnitude and phase components respectively 
of the (D{/})^ term of Equation (13), calculated by using the relationship in Equation 
(19). Similarly, Figs. 2D and 2E show the magnitude and phase components respectively 
of the /D^{f} term of Equation (13), calculated by using the relationship in Equation (20). 

15 The magnitude and phase components respectively of the complex energy operator Tc is 
shown in Figs. 2F and 2G. Referring to Equation (11), the phase component of the 
complex energy operator has double the orientation angle j3o of the fringe pattern / 

All the above calculations and derivations may alternatively be implemented as 
convolution operators without Fourier transformation. 

20 However, when the energy operator 4^c is applied to a fringe pattern /containing 

noise, such as that shown in Fig. 3A, the resulting magnitude and phase components of 
the complex energy operator 4^c, shown in Figs. 33 and 3C, are less satisfactory. The 
primary reason for the degradation is the linear (conical) spectral enhancement of the 
differential operator D. The differential operator D effectively enhances high frequency 

25 noise, as it essentially operates as a high-pass filter. In particular, in the areas of low 
spatial frequency Qq, such as the area 20 on Fig. 3A, the contribution by the noise signal 
due to the differentiation dominates the determination of the orientation of the fringe 
pattern. Referring to Fig. 3C, the phase component, from which the orientation can be 
obtained by using the relationship from Equation (11), provides near random noise in the 

30 corresponding area 22. 
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One possible correction is to apply a low pass filter function to the input image or 
the result. However, the energy operator 4^ is not a linear (shift invariant) operator. It is 
dependent on products of the input function / and its derivatives df/dx and d^f/dx\ as can 
be seen from Equation (1). Low pass filtering results in the image becoming blurred. 
5 To overcome the deficiencies of the complex energy operator while maintaining 

the ease of calculation of the orientation angle Po, according to the preferred embodiment, 
a pure phase operator D^, thus having uniform (spectral) magnitude, is created by setting 
q in Equations (19) and (20) to unity, to define the following Fourier transform pair: 

exp(/<p)F « {/(x, y)} (2 1 ) 

10 The pure phase operator is a spiral phase (or vortex) operator which results in 

the removal of noise without a blurring of the signal. By applying the modified energy 
operator defined as: 

'^M = {D,{f]Y-fDl{f} (22) 

to the image in Fig. 3A, the magnitude and phase components respectively of the 
15 modified energy operator shown in Figs. 3D and 3E, are obtained. The comparison 
of the phase components of the complex energy operator shown in Fig. 3C and 
modified energy operator shown in Fig. 3E, and in particular to the corresponding 
areas 22 and 24 respectively, indicate that the modified energy operator 4^m provides a 
better result. 

20 It is noted that, similar to the energy operator T, the modified energy operator 4^m 

is also non-linear. A consequence of this is that it is generally not possible to obtain the 
performance of the modified energy operator by merely applying alow pass fiUer to 
the input function, nor by filtering the output function, of the energy operator 4^. 

The modified energy operator has a property of scale invariance, which it 

25 inherits from the scale invariant Dm operator. Scale invariance is another aspect of the 
operator's spectral neutrality and means that the operator essentially gives outputs 
independent of feature size. 

To further illustrate the advantages of the modified energy operator 4^m, the 
complex energy operator Tc and the modified energy operator Tm are respectively 

30 applied to an image with uniform noise in the range ±64 greylevels shown in Fig. 4A. 
The magnitude and phase components respectively of the complex energy operator are 
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shown in Figs. 4B and 4C, while the magnitude and phase components respectively of the 
modified energy operator are shown in Figs. 4D and 4E. Because of the high level of 
noise in the input image, the phase component of the complex energy operator is 
completely weighed down by the contribution of the noise. The modified energy operator 
gives a much clearer phase without any obvious blurring that would be expected of 
noise reduction by conventional low pass filtering of the complex energy operator 

Another comparative example wherein the performance of the modified energy 
operator can be seen is shown in Figs. 5A to 5E. Fig. 5A shows an amplitude and 
frequency modulated image. The magnitude and phase components respectively of the 
complex energy operator are shown in Figs. 5B and 5C, while the magnitude and 
phase components respectively of the modified energy operator Tm are shown in Figs. 5D 
and 5E. The complex energy operator again has difficulty in regions of low spatial 
frequency 7^, such as area 30, causing the corresponding area 32 in Fig. 5C to have high 
levels of uncertainty. On the other hand, the corresponding area 34 when the modified 
energy operator is applied, provides a much clearer result. 

It is fiirther noted that since the underlying fringe pattern of the images in Figs. 3 A, 
4A and 5A are the same with different processes applied thereto, the processes not 
affecting the orientation of the fringes of the underlying pattern, the results of the 
modified energy operator Tm shown in Figs. 3E, 4E and 5E are substantially similar. 

The modified energy operator 4^m may be implemented entirely in the Fourier space 
domain by using complex convolution kernels that correspond to the pure phase spirals. 
With the kernel k and its Fourier transform K, the Fourier transfonn pair can be defined 



as: 



K{u, v) = expO^) . « . . ^(,,^) (23) 

wherein the spatial polar coordinates satisfy the following: 



X = rcos{0) 
y = r sin(^) 



(24) 



The convolution kernel k reduces by the inverse square of the radius, and also has a 
spiral phase structure, but rotated by 90° with respect of the x-y coordinates. According 
to the above, since it is now known those operations required to be performed to extract 
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local orientation information from a fringe pattern, it is not necessary to perform such 
calculations in the Fourier domain. Rather, the desired functions may be recalculated in 
the real domain thereby facilitating ease of processing. In many cases the desired 
functions may be approximated by suitably truncated kernels. The inverse square drop- 
off allowing relatively small truncation errors even for small kernels 

Referring again to Fig. 1, by considering a sufficiently small region Q of images 
composed of line or edge features, the local structure fits the model of Equation (2). This 
is true for many image features, the most obvious exceptions being textures that contain 
multiple AM-FM modulated signals. Sometimes an image may approximate the model of 
Equation (2) except for a low or zero frequency background offset in intensity. In such 
cases this background level may be removed by high pass filtering (or related methods) 
leaving an image that does satisfy Equation (2) which can then be processed using the 
complex energy operator formalism. 

Where Equation (2) is representative, the modified energy operator Tm, as defined 
in Equation (22), to be used for the calculation of the orientation angle Po of all such 
images. 

An example of such an image and the operation of the modified energy operator 4^m 
are shown in Figs. 6A to 6E. Fig. 6A shows a texture image. The texture image is 
characterised by distinct lines 40 and areas of low variation 41. The magnitude and phase 
components respectively of the complex energy operator are shown in Figs. 6B and 
6C, while the magnitude and phase components respectively of the modified energy 
operator are shown in Figs. 6D and 6E. The phase component of the modified energy 
operator *4^m provides a result that is more intuitively representative of the orientation of 
the texture miage of Fig. 6A. 

A flow diagram of a method of estimating the pattern and texture orientation of 
images is shown in Fig. 8. An input image is inserted in step 200, The input image is 
typically digital format with a predetemiined number of columns and rows. Alternatively 
the image may be scanned to convert it into the digital format. 

Although the preceding analysis has been in terms of continuous functions and 
continuous operations, the analysis is also directly applicable to discretely sampled 
functions (images) and discrete operations. 
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A Fourier transform of the image is calculated in step 210, typically using the 
Fast Fourier Transform (FFT). The Fourier transformed image is multiplied with a term 
exp(i(p) in step 220. 

An output of step 220 is again multiplied with the term exp(i(p) in step 230 
before it is inverse Fourier transformed, typically using an Inverse Fast Fourier Transform 
(IFFT) in step 240. The result is then multiplied with the input signal in step 270. 

The output from step 220 is also inverse Fourier transformed in step 250 before it 
is squared in step 260. The output from step 270 is subtracted from the output of step 260 
in step 280. From this complex signal, the angle of the signal is calculated in step 290 
'before it is halved in step 300 to provide an orientation angle f5o as an output in step 310. 

The method of estimating the pattern and texture orientation of images is 
preferably practiced using a conventional general -purpose computer system 100, such as 
that shown in Fig. 7 wherein the calculation of the orientation angle pQ may be 
implemented as software, such as an application program executing within the computer 
system 100. In particular, the calculation may be effected by instructions in the software 
that are carried out by the computer. The software may be stored in a computer readable 
medium, including the storage devices described below. The software is loaded into the 
computer from the computer readable medium, and then executed by the computer, A 
computer readable medium having such software or computer program recorded on it is a 
computer program product. The use of the computer program product in the computer 
preferably effects an advantageous apparatus for demodulating two-dimensional fringe 
patterns in accordance with the embodiments of the invention. 

The computer system 100 comprises a computer module 102, input devices such 
as a keyboard 110 and mouse 1 12, output devices including a printer^lt)8 and a display 
device 104. 

The computer module 102 typically includes at least one processor unit 114, a 
memory unit 118, for example formed from semiconductor random access memory 
(RAM) and read only memory (ROM), input/output (I/O) interfaces inckiding a video 
interface 122, and an I/O interface 116 for the keyboard 110 and mouse 112. A storage 
device 124 is provided and typically includes a hard disk drive 126 and a floppy disk 
drive 128. A magnetic tape drive (not illustrated) may also be used. A CD-ROM 
drive 120 is typically provided as a non-volatile source of data. The components 114 
to 128 of the computer module 102, typically communicate via an interconnected bus 130 
and in a manner which results in a conventional mode of operation of the computer 
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system 100 known to those in the relevant art. Examples of computers on which the 
embodiments can be practised include IBM-PC's and compatibles, Sun Sparcstations or 
ahke computer systems evolved therefrom. 

Typically, the application program of the preferred embodiment is resident on 
the hard disk drive 126 and read and controlled in its execution by the processor 114. 
Intermediate storage of the program may be accomplished using the semiconductor 
memory 118, possibly in concert with the hard disk drive 126. In some instances, the 
application program may be supplied to the user encoded on a CD-ROM or floppy disk 
and read via the corresponding drive 120 or 128, or alternatively may be read by the user 
from a network via a modem device (not illustrated). Still further, the software can also 
be loaded into the computer system 100 from other computer readable medium including 
magnetic tape, a ROM or integrated circuit, a magneto-optical disk, a radio or infra-red 
transmission channel between the computer module 102 and another device, a computer 
readable card such as a PCMCIA card, and the Internet and Intranets including email 
transmissions and information recorded on websites and the like. The foregoing is merely 
exemplary of relevant computer readable mediums. Other computer readable mediums 
may be practiced without departing from the scope and spirit of the invention. 

The calculation of the orientation angle Po may alternatively be implemented in 
dedicated hardware such as one or more integrated circuits performing the functions or 
sub functions of the calculation. Such dedicated hardware may include graphic 
processors, digital signal processors, or one or more microprocessors and associated 
memories. 

4 « 



D1P02 CFPI682AU 



I:\ELEC\CISRA\DIP\DtpO2V478580.doc 



- 13 - 

^ ^h e ^C laims defining the invention are as follows 



1. A method of estimating an orientation angle of a pattern in an image, said 
method comprising the steps of: 

applying a G0i?nplex #n^^^^^^ to the image to provide-an energy encoded 

image; 

determining a phase component of said energy encoded image; and 
calculating said orientation angle from said phase component of said energy 
encoded image. 

2. A method as claimed in claim 1 wherein 
said complex energy operator being defined as: 

'^c{f] = (D{f})'-fD^{f} 

and said phase component of said energy encoded image being defined as: 
2^o=arg(4',{/}) 

3. A method as claimed in claim 1 wherein 

said complex energy operator being a modified complex energy operator defined as: 

and said phase component of said energy encoded image being defined as: 
2y9o=arg(4',,{/}) 

4. A method as claimed in any one of claims 1 to 3 wherem said image is 
preprocessed to remove background offsets. 

5. Apparatus for estimating an orientation angle of a pattern in an image, said 
apparatus comprises: 

means for applying a complex energy operator to the image to provide an energy 
encoded image; 

means for determining a phase component of said energy encoded image; and 
means for calculating said orientation angle from said phase component of said 
energy encoded image. 
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6. Apparatus as claimed in claim 1 wherein 
said complex energy operator being defined as: 

^c{/}=(^{/})'-yz)M/} 

and said phase component of said energy encoded image being defined as: 

7. Apparatus as claimed in claim 1 wherein 

said complex energy operator being a modified complex energy operator defined as: 

10 and said phase component of said energy encoded image being defined as: 
2^o=arg(T^{/}) 

8. Apparatus as claimed in any one of claims 6 to 8, said apparatus further 
comprising a means for preprocessing said image to remove background offsets. 

15 

9. A method of estimating an orientation angle of a pattern in an image, said 
method being substantially as described herein with reference to the drawings. 

10. Apparatus for estimating an orientation angle of a pattern in an image, said 
20 apparatus being substantially as described herein with reference to the drawings. 



DATED this Fourteenth Day of April, 2000 
Canon Kabushiki Kaisha 
25 Patent Attorneys for the Applicant 

SPRUSON & FERGUSON 
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Fig. 5D Fig. 5E. 
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